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FOREWORD 



This research and development was performed within subproject Z1175-PN.05 
(Improved Effectiveness in Course Design, Delivery, and Evaluation) and under the 
sponsorship of the Deputy Chief of Naval Operations (Manpower, Personnel, and Training) 
(OP-01). The objective of the subproject was to develop an empirically-based instruc- 
tional design support system to aid developers in choosing instructional alternatives based 
on cost/benefits and specific resource limitations. The purposes of the effort described 
herein were to identify the problems^ that authors at the Navy's Instructional Program 
Development Centers (iPDCs) had in using the Instructional Systems Development model 
for developing instruction, determine what authoring aids were available or needed to 
assist them in developing instructional materials, and assess tri-service availability and 
utilization of authoring aids. 

Results are intended for use by the Chief of Naval Education and Training 
(specifically, the IPDCs) and the Chief of Naval Technical Training. 



JAMES F. KELLY, JR. 
Commanding Officer 



JAMES W. TWEEDDALE 
Technical Director 



SUMMARY 



Problem 

Instructional development in the Navy's Instructional Program Development Centers 
(IPDC?5 is guided by the Interservice Procedures for Instructional Systems Development 
(ISD). Recent studies on the use of the ISD model for instructional development suggest 
that ISD does not provide complete enough guidance for relatively inexperienced authors 
who design and develop training programs. More experienced personnel are not readily 
available to the Navy and the cost of upgrading instructional stcills of the present IPDC 
staff would be very expensive. However, before the present ISD pVocess can be improved, 
specific information is needed about the difficulties encountered in authoring instruction. 

Objectives 

The objectives of this effort were to (1) identify the problems that authors at IPDCs 
have in implementing the ISD model, (2) determine what other authoring aids exist that 
could be used to assist authors in developing instructional materials, (3) assess the need 
for additional aids at the IPDCs, and W assess tri-service availability and utilization of 
authoring aids. 

Approach 

This effort was conducted in two phases. In the first phase, conducted in 1978, a 
structured questionnaire was developed and administered to IPDC staff members to 
determine if they believed that existing procedures were adequate for instructional 
development. Two approaches were used in the second phase, which was conducted 2 
years later after IPDC personnel had increased experience in the use of the ISD process. 
First, to extend results of the questionnaire administered during Phase 1, IPDC personnel 
were extensively interviewed to obtain more detailed information regarding difficulties 
encountered in developing training using the ISD process. Second, tri-service agencies 
were contacted to determine what authoring aids were available, whether they recognized 
the need for authoring aids, and whether any were being developed. 

Results 

Questionnaire responses from Phase I revealed that improved or new techniques were 
needed in four areas: (1) methods for evaluating appropriateness of training objectives, 
(2) methods for developing tests and test items, (3) procedures for evaluating objective 
quality, and W quality control procedures for the instructional development process. 

These results helped define R&D needs and led to the development of the Instruc- 
tional Quality Inventory (IQI) as an aid for design and development. The individual 
interviews held in Phase 2 revealed that, although some of the problems uncovered in 
Phase 1 had been dealt with fairly adequately by IQI, problems still existed in (1) selecting 
instructional strategies, (2) acquiring knowledge about th^ subject matter, (3) writing 
terminal and enabling objectives, W writing test items, and (5) adhering to the ISD model 
because of time and system constraints. In addition, information was obtained about the 
education and experience level of IPDC authors. 

Contacts with tri-service agencies revealed that authoring aids being developed were 
(1) service specific, (2) still in the development stage, and (3) not ready for modifications 
that would meet Navy application needs. 
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Cone lusions 



Development of instructional nnaterials and test construction were difficult areas for 
most IPDC authors using I5D procedures. In view of the complexity and length of training 
needed to provide skilled instructional designers, and the lack of such people among the 
IPDCs, improved guidance is needed for the process of materials development. 

Recommendations 

1. Authoring aids already available, such as the IQI and the Navy's Author Training 
Course, should be adapted for use as on-line computer-based aids. 

2. NAVPERSRANDCEN should develop an author-aiding system that utilizes exist- 
ing aids and incorporates new ones as needed. The system should be transportable so that 
it can be widely utilized by groups developing Navy instruction in various locations. It 
would be available for tri-service utilization. 

3. IPDC staff members should be strongly encouraged to take advantage of 
available instructional technology courses and, whenever possible, time should be set aside 
for this training. 

^. Coordination should be maintained with the appropriate tri-service agencies to 
facilitate interservice exchange, as well as eventual application of new authoring aids. 
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INTRODUCTION 



Problem and Background 

The Navy depends heavily upon its training system to provide personnel who are fully 
equipped to handle the high technology requirements of the modern Navy. The training 
system, in turn, relies upon performance-based instruction, which is derived from job 
specification and training analysis. Appropriate instructional procedures are then 
developed to prepare students to meet job requirements. 

Instruction for a variety of Navy technical schools is developed and revised by the 
Instructional Program Development Centers (iPDCs), San Diego and Great Lakes (later 
Memphis and Pensacola), which were established as central instructional organizations by 
the Naval Education Training Command in 1976. Developing and revising instructional 
programs is a complicated process that requires either highly educated and experienced 
designers and developers or, alternatively, a set of "complete" guidelines that enables less 
experienced persons to do the work (Hodak, Middleton, & Rankin, 1979; Miller, Swink, <5c 
McKenzie, 1978). Since highly educated and experienced designers and developers are 
rare, the Interservice Procedures for Instructional System Development (ISD) (Branson, 
Rayner, Cox, Furman, King, & Harnum, 1975) were designed and developed to serve as the 
basis for instructional development in the three services. The ISD model is, in effect, a 
design aid describing what has to be done and was intended to provide guidance for how to 
do it. One of the implicit assumptions behind its adoption by the services is that 
techniques and procedures needed to develop instruction are specified in a way that allows 
military subject matter experts (SMEs) with little or no background in education or 
instructional technology to design training materials. 

The ISD model is an application of the systems analysis approach to training. This 
approach was first applied to training development in the 1950s (Churchman, 1968). 
Developers who were familiar with the development procedures and had extensive 
knowledge about the effectiveness of various training methods prepared guidelines that 
would enable others to use these procedures and methods. The notion underlying the 
development of these guidelines, which were intended to provide sp)ecific, step-by-step 
procedures for designing and developing instruction, was that standardized output of good, 
usable instruction would result. 

However, recent studies on the use of the ISD model for instructional development 
suggest that ISD procedures do not provide untrained, inexperienced personnel with 
enough guidance through the design/development process or at the critical decision points 
to guarantee good instructional program development (O'Neal, Faust, & O'Neal, 1979). 
For example, Vineberg and Joyner (1980), as well as Hodak et al. (1979), studied the 
institutionalization of ISD and found that the procedures were difficult to use, as well as 
unreliable. Too much is still left up to the individual author in translating ISD from a 
series of "what to do" steps to an integrative process of instructional development 
(Montmerlo & Tennyson, 1976; Montmerlo, 1979). Further, in empirical evaluations of the 
implementation of the ISD process in the military, significant problems were found (e.g., 
guidance was incomplete, steps were difficult to relate to one another, and job analyses 
were incomplete) (Vineberg & Joyner, 1980; Rickertson, Schultz, & Wright, 1970; Hodak et 
al., 1979; Miller et al., 1978). Also, recent examination of implementation problems in 
education suggest that a serious problem in effecting innovations in teaching practice is 
the lack of control over the details of implementation (i.e., the lack of quality control, 
monitoring systems, and effective, easy to use guidelines) (Aagard & Braby, 1976; 
Bunderson, 1977: Upson, 1973; Van Pelt & Rich, 1975). 
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At the t'nne of this research, many of IPDCs civilian instructional technologists, who 
were expected to develop and maintain expertise in curriculum design and development, 
were relatively inexperienced, and their rate of turnover was high. Further, the ISD 
procedures designed to guide the instructional development process appeared to be 
Incomplete and allowed variability (cf. Rickertson et ah, 1970). More experienced 
personnel are not readily available to the Navy and the cost of upgrading instructional 
skills of the IPDC staff would be very expensive. Therefore, procedures must be refined 
and developed to reduce the variability allowed by the present design and development 
guidelines. 

One possible solution could be to revise or develop new automated aids that would 
more adequately support relatively inexperienced authors in producing instructional 
materials as well as provide a method for assuring quality control in the implementation 
phase. Another would be to provide improved training for authors to upgrade their skills 
in developing instructional materials. However, before the present ISD process can be 
improved, specific information is needed about the difficulties authors encounter and the 
type of authoring aids that are available. 

Objectives 

The goals of this effort were to (1) identify problems authors at the IPDCs have in 
implementing the ISD model, (2) determine what authoring aids in addition to ISD are 
being or could be used to assist authors in developing instructional materials, (3) assess 
the need for additional aids for use at the IPDCs, and W determine tri-service availability 
and utilization of authoring aids. 

METHOD 

This effort was conducted in two phases. In the first phase, conducted in 1978, the 
ISD procedures in use at the IPDCs were reviewed and a flowcha/t made of the activities, 
personnel responsibilities, and information flow in the ISD design and development phases 
(Phases II and III) (see appendix). In the process of producing the flowchart, 1^ design and 
development tasks or decisions critical to the success of the overall procedures were 
identified. These 1^ areas were used to construct a structured questionnaire designed to 
determine if the existing procedures in these areas were adequate or if new procedures 
should be developed. The questionnaire was administered to 57 development personnel— 
32 at IPDC San Diego and 25 at IPDC Great Lakes. 

The second phase was conducted in 1980, after IPDC personnel had increased their 
experience in the use of the ISD process. In thiS phase, issues were considered that were 
related to the adequacy of ISD procedures and the need for revision or development of 
additional authoring aids. These issues were (1) authoring procedures currently in use at 
the IPDCs, and (2) an assessment of tri-service utilization of authoring aids. 

To assess the authoring procedures then in use and to determine potential training 
requirements, extensive interviews were conducted with IPDC personnel at San Diego and 
Great Lakes who were very fanriiliar with both the ISD model and the problems with the 
model at IPDC. This included members of the administrative staff, designers, writers, 
and editors.' This effort extended the results of the structured questionnaire administered 
to IPDC staffs during Phase 1. 
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The areas of concern in these interviews were (1) authoring aids currently in use, (2) 
education/experience levels of authors at IPDC, and (3) problem areas in curriculum 
development. Several questions needed to be answered in these areas before training 
recommendations could be made. To reduce bias due to individual perception, information 
gained during the interviews was confirmed with at least three other interviewees before 
being included in this study. Recommendations concerning the requirements for addi- 
tional authoring aids needed to support specific blocks in the ISD model were then 
formulated. 

To determine what authoring aids were available and in use, a survey was made of the 
appropriate agencies in the Army, Air Force, and Navy. Civilian contractor firms working 
on government contracts in instructional development were also surveyed. 

RE5JLTS 

Phase 1 

Table 1, which presents questionnaire results, shows that the majority of respondents 
indicated that new or improved procedures would be useful in 13 of the 1^ areas (all but 
No. 5, which was reported as "not applicable" by 26% of the respondents). The areas 
identified as most in need of new or improved methods were (1) methods for evaluating 
the appropriateness of training objectives (No. 1), (2) methods for developing tests and 
test items (No. 2), (3) procedures for evaluating objective quality (No. 8), and (^) quality 
'control procedures for the instructional development process (No. 13). The objective 
adequacy assessment portion of the IQI (Wulfeck, Ellis, Richards, Merrill, <5c Wood, 1978; 
Ellis, Wulfeck, & Fredericks, 1979) partially addresses the first three areas listed and 
completely covers the latter one. 

IPDC personnel identified a need for comprehensive procedures for making decisions 
about testing and for writing criterion-referenced test items that are consistent with 
objectives. Vineberg and 3oyner (1980), in their review of ISD, also identified the need for 
better test development procedures. They recommended that "ISD models should be'^ 
expanded to provide procedures for identifying and maintaining congruence (i.e., consis- 
tency) between the behavioral requirements of objectives and test items." 

Phase 2 

Authoring Procedures Used at the IPDCs 

Authoring Aids Currently in Use . The IPDC staffs at Great Lakes and San Diego 
agreed that existing authoring aids did not provide adequate guidance for producing 
standardized and usable instructional materials. At the ►time of this effort, there were 
four primary aids that provided some degree of guidance for instructional program 
development. (These are still the primary guidance documents.) 

1. NAVEDTRA 110 (CNET, 1979), a procedures manual that was designed primarily 
for use by management in monitoring steps in the instructional process rather than to 
provide guidance in design and development.^ 



A substantial revision of this manual was undertaken in 1981. This revision includes 
more guidance for design and development. 
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Table 1 



3. 



S. 
9. 

10. 
11. 

12. 
13. 
U. 



Responses of IPDC Personnel to Questionnaire on Need for lmproved ISD Procedures 



Percent Responding 



Area 


Not appli- 
cable 


Current pro- 
cedures 
are adequate 


No need 
for this 


Probably 
useful 


Extremely use- 
ful, would 


N 


More systematic methods for evaluat- 
ing the appropriateness of objectives 
for developing training materials. , 


5.26 


1-75 


0 


36.85 


56. 1^* 


57 


Methods for developing test items 
from objectives, including per- 
formance tests. 


5.26 


5.26 


1.75 


33.33 


54.40 


57 


A test specification plan including 
item format, number of items, and 
test writing instructions. 


7.15 


5.35 


7.15 


^^8.21 


32.14 


56 


Procedures for making tests 
diagnostic, or methods for develop- 
ing diagnostic tests. 


9.09 


3.65 


9.09 


^^0.00 


38.17 


55 


Methods for deciding whether entry 
tests and pretests are cost 
effective. 


26.32 


3.51 


21 .05 


2U.56 


24.56 


57 


AietnoQs tor sequencing cinu oli ui_iui 
ing objectives. 


8.93 


7.U 


1.78 


35.72 


46.43 


56 


Procedures for planning the manage- 
ment of instruction: 

a. Structuring the course into 
lessons and modules. 

b. When to test and how much to 
test. 

c. Managing or controlling student 
progress. 

d. Cost-effectiveness of different 
management options. 


12.28 
10.53 
17. 5i* 
2^*.57 


U.O^* 
8.77 

10.53 
1.75 


1.75 
5.26 
5.26 
5.26 


^^5.61 
38.60 
^3.86 
36.8^ 


26.32 
36.84 
22.81 
31.58 


57 
57 
57 
5^ 


A procedure for evaluating the 
quality of objectives. 




5.56 


0 


31.^8 


61.11 


54 


More systematic procedures for 
specifying to authors what should 
be included in a lesson; for example, 
kinds and numbers of examples, 
practice items, etc. 


9.26 


5.56 


9.26 


35 . 19 


4U . / *f 




Criteria for deciding among types 
of instructional media. 


9.09 


25.^6 


7.27 


32.73 


25.45 


'\ 's 

J J 


Media for evaluating existing 
materials, and ways to decide 
if it is worth revising them. 


3.70 


9.26 


5.56 


35.20 


46.29 


54 


More systematic procedures for 
writing instruction. 


9.26 


7.^*1 


7.^1 


33.33 


42.59 


54 


Quality control methods for the 
instructional development process- 


0 


5.56 


1.85 


35.19 


57.41 


54 


Methods for validating instruction 
from student trials. 


12.96 


16.67 


1.85 




24.08 


54 
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2. NA^EdTRa i06A (Branson et al., 1975), the Navy»s ISD manual, which was 
design^ to provide guidance specific to ISD procedures at the designer/developer level. 
This rO^^VsaU as ^v^ll as NAVEDTRA 110, is lengthy and somewhat cumbersome. Also, 
someo^ ^he guidance contained therein is incomplete. 

3. Yh^ iQl (Wulfeck et al., 1978), which is relatively short and has a workbook that 
orients ^he writer to ^he IQI concepts and provides practice in using those methods (Ellis, 
Wulfeck, «c Fredericks, 1979). 

^. Th^ Author Training Course (ATC) (NAVEDTRA 10003), which was designed as a 
series self-instructional manuals for the development phase of ISD. Although a few 
IPOC 5tajf mernbers had attended a workshop, such ATC workshops were not offered in a 
systerri^Uc n^anner nor were they available to the entire staff. Individual staff members 
had fa^n^ various tests and manuals that provided some guidance, but neither the 
authoring aids nor the training Were available to the staff generally. Finally, since there 
was n^^ ^ecific training or skill-updating schedule for the staff, additional skills were 
acquir^ only through personal motivation (i.e., additional coursework, information 
sharing between staff > pertinent reading). 

E^UQati^n/^x^e^ One of the problems at the iPDCs was the wide variability in 

the st^^f rtiernbers' technical preparation for designing/writing instruction. Only a few 
staff rnervi^ers h^d specialized, advanced training in instructional design. For example, at 
one IPDC surveyed, rnost of the basic staff consisted of English, linguistics, and education 
major5? retired journalists and teachers. Approximately 60 to 70 percent of the 
design^^s/v^r iters were young with Bachelor or Master of Arts degrees, and many of these 
were in t^eir first professional job. There were also a number of retired military who had 
been cl^^si^^^ education specialists, and a few active duty military who received an 
ISD orientation upon their arrival at IPDC. 

fsAosx of the staff were introduced to instructional design, job/task analysis, and 
writing instructional materials by on-the-job training at IPDC. Given this lack of 
specific training in ISd and the paucity of author training aids available for use, the 
training of incoming staff had relied almost totally upon information, tools, and 
techni^^^s Provided by the more experienced staff members. This condition produced 
nonstaf^d^r^l^ed approaches to the analysis, design, and production of materials. 

A't ^ach iPbCj there was an evaluation group tasked with standardizing the 
currictJlUtn output, aS Well as a separate editorial staff tasked with standardizing writing 
styles' Ho^^ver? because of the general lack of training in basic instructional design and 
writings th^ initial output tended to be relatively poor, requiring extensive rewrite and 
even red^gign before a final version of the curriculum could be produced. As a result, the 
evalua^i^n staff was overburdened. 

problem Areas in Curriculurn Development. The IPDC staff members interviewed 
felt that writers/developers needed training in several areas. Probably the most 
signlfi^ar>|-|; area of concern was the selection and development of instructional strategies 
(i.e., the v^ays in v^hich the content is arranged and presented to teach to the course 
objectives)* System constraints within IPDC and at the instructional site limited choices 
of hov^ ^^st to teach the materials. These contraints included the costs of audiovisual 
instru<-^i^n^^ n^^dia (filnns, slides, etc.), availability of simulated equipments, and 
availat>iUty of tools and materials needed for hands-on training, as well as internal IPDC 
problems of personnel, graphics requirements, and availability of SMEs. There was, 
howev^^ another problem at the IPDC staff level in selecting the appropriate instruc- 
tional s^ta^^gy (^ithin the known constraints) for satisfactorily teaching the required job 



performance: The ability to select appropriate strategies was somewhat restricted by the 
limited knowledge about instructional development held by most of the staff rnembers. 
For example, because the use of graphic illustrations as an instructional tool ^^s riot well 
understood by the staff, many instructional strategy decisions were rn^d^ without 
considering the entire range of possibilities. 

The difficulties found *n selecting appropriate instructional strategies ^^^re further 
complicated by lack of knowledge about the subject area. Designers/v^^ritefS relied 
heavily upon individual SMEs and the curriculum from the existing course for the required 
knowledge. Before and during the development of the^ instructional rtiat^rials, the 
designer reviewed the previous course material, assisted by (usually) a single sME. The 
problem for the designer was how to use this SME effectively in determining ^^hat should 
be in the course. The process of extracting valid information from an SME 15 b^th critical 
and complicated. Depending upon the technical level of the course requirements, 
incomplete understanding of the relationship between critical tasks and job Performance 
can lead to oversimplification, incorrect prioritization of critical task5> poorly 
defined terminal objectives. Although the ISD analysis phase is supposed to define tasks 
adequately, feedback from designers, writers, and evaluators indicated that this was a. 
problem area. 

There v/as general consensus among those interviewed that personnel re<l^i^ed further 
training in writing terminal and enabling objectives. Although some training had been 
given in this area, apparently either the examples in texts and workshops were not 
applicable to the IPDC environment, or the teac'ung was not adequate enough to enable 
the authors to transfer the knowledge to their wwrk. It was suggested that a Workshop be 
structured around course material in which development problems had been et^countered 
so that the workshop would be more relevant to the IPDC. In additic^n, there ^as 
apparently some difficulty in understanding the relationship between enabling objectives 
and the terminal objective. 

Training in writing test items was also needed. At the time of the interviews, there 
was no expert in test construction on the IPDC staff. The type of test jse^ to assess 
performance in IPDC-developed instructional programs is called a criteri^^n^referenced 
test. In such tests, an individual's performance is compared (referenced) to absolute 
standard or criterion of what a person must know or be able to do. Specific training in 
scoring and interpretation of test results is required for making appropriate decisions 
about student performance. 

The criterion-referenced test is very different from the better knc^^n and more 
commonly used norm-referenced test, in which each student's performance i^ compared 
and ranked relative to the performance of other students. Norm -referenced tests are 
used to help make decisions about promotions, about college entrance exarnir^ations, and 
in other situations where the aim is either to select the best possible studet^ts above a 
cutoff line or to fill a single job opening. Since the object in a training 5itUation is to 
train all students to a specific performance objective rather than to s^^^c^ a gWen 
percentage, the most positive outcome of a training environment would b^ to have 100 
percent of the students meet the performance objective. In addition, there r\o separate 
norming or evaluation of the tests; the tests are validated at the same time as the course 
materials, because of such system constraints as lack of studerit subjects, ti^^, and staff 
experts to evaluate the results. 

When questioned about the degree of adherence to the ISD model iri terms of 
completeness and sequence, respondents indicated that the model wa5 not strictly 
followed. The main reason seems to be that various components in the Navy system do 
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not allow it. For example, the design phase was often begun before CNET approval of the 
ISD analysis was obtained. Design decisions were made on the basis of logical guesswork 
that predicted the most likely material to be included in the final package. Since these 
decisions affected the construction of the next ISD steps, a considerable amount of 
redesign would have been required if the predicted and approved final packages differed 
significantly. 

Another system constraint was the fact that IPDC did not have final control over all 
of the resources it used. This lack of control required a certain amount of flexibility in 
following the sequence of the L^D model. For example, SME assignrnents and procurement 
of needed field equipment were not under IPDC jurisdiction. The particular timing of 
their availability influenced the sequence of tasks. In addition, procurement procedures 
frequently resulted in delays in acquisition of planned audiovisual or other training 
materials, which may have caused changes in the original instructional design. In 
summary, the ISD model was often compromised by both economic and logistical 
contraints. 

Assessment of Tri-service Utilization of Authoring Aids 

Initial contacts with appropriate agencies in the three services indicated that, 
although they were interested in the author aid development problem? there was limited 
coordination of efforts among groups. The Air Force Human Resources Laboratory had 
developed a research and development plan, but funding cuts had an but eliminated their 
participation. However, under contract, authoring support software was developed. With 
this software, simulation scenarios could be prepared using conversational dialogue 
instead of a programming language so that the authoring system could be used by 
personnel not fluent in computer programming. Also, computer-based decision aids for 
writing learning objectives were developed. 

In the Navy, the Navy Personnel Research and Developmerit Center (NAVPERS- 
RANDCEN) began the development of computer-based aids for instructional design in 
1982. The first phase of this effort is intended to help designers at critical points in the 
ISD process by guiding and monitoring each step involved in using the IQI and in 
developing tests. Programs will then be expanded gradually to provide specific guidance 
on accomplishing each task, allow access to relevant data bases (e.g., test-item files, 
classifications of objectives, etc.), and help select alternative forms of instruction. 

Another Navy system, which is being developed by the Navy Training Analysis and 
Evaluation Group (TAEG), can be used by authors who are SM^s to write, edit, and 
produce programmed training materials (Braby, 1979). In this system, the author types 
requests for information about materials and test items on a terminal that is connected to 
an editing and production system. The system then arranges the material in a format 
appropriate for programmed instruction and produces the text. Not only is the time 
required for authoring and producing materials reduced but also, and more importantly, 
the requirements for instructional design expertise. Such systems will enable personnel 
with only modest experience in instructional design and development to develop quality 
instructional materials. 

\ Two major efforts were being conducted under contract to the Army Research 
Institute (ARI). The first is focused on completing off-line aids for all blocks in the ISD 
model (Schultz, Hibbits, Wagner, & Seidel, 1979). However, since all of these aids are 
Army military occupation specialty (MOS) specific, they are not readily applicable to tri- 
service needs. The second effort, which is being monitored by the aRI Field Unit located 
at Ft. Bliss, concerns the development of a five-part model of a computer -based authoring 



aid. The five parts are (1) Training Prescription, (2) Training Hierarchy, (3) Cost 
Partition, (*) Test Analysis Partition, and (5) Training Characteristics Partition. With this 
model, an author can input information, and the computer will supply a training hierarchy 
that includes media method selection and training criticality. 



DISCUSSION AND CONCLUSIONS 

It is clear from the results of both phases of this effort that there were problems in 
the implementation of ISD. Results from the structured questionnaire administered 
during Phase 1 indicated^hat the areas most in need of improved or new authoring aids 
were (1) methods for evalC^ting tiie appropriateness of training objectives, (2) methods for 
developing tests and test items, (3) procedures for evaluating objective quality, and W 
quality control procedures for the instructional development process. The results from 
Phase 2, conducted 2 years later, showed that some of the previous problems had been 
dealt with fairly adequately by using IQI (particularly those with quality control 
procedures). However, IPDC staff members rioted that additional aids were needed in the 
areas of (1) instructional strategy, (2) depth of subject area knowledge, (3) media 
selection, W terminal/enabler objective writing, and (5) test construction. 

Since the area of test construction, which was identified as a problem area in both 
phases, appeared to be a very difficult area for the staff at the IPDCs, NAVPERSRAND- 
CEN has developed an improved test development procedures. The results of the effort- 
are included in a handbook of "job-aids" for Navy -tesfiieVelopers that is being compiled 
from a variety of sources (Merrill & Wood, 197'f; Swezey & Pearlstein, 197^^; Roid & 
Haladyna, 1979; Frederickson, Smith, & Pearlstein, 1979). The handbook was refined 
during training workshops for Navy personnel and a final version was forwarded to CNET 
in FY82. 

The level of education and experience of IPDC staff members was less than optimal 
for producing high-quality instructional materials. The fact that in-service training had 
been placed on a much lower priority than product output made it difficult for th^ authors 
to upgrade their writing/developing skills. 

There are a variety of training solutions that could be effective in overcoming the 
training problems identified at IPDC. Those potential solutions include intensive traihing 
of the IPDC staff in instructional technology, developing new authoring aids, or 
automating existing authoring aids. A system of computer-based authoring aids could 
facilitate the design and development phases of the ISD process by providing specific 
guidance to authors as they progress through each ISD block (Taylor, 1979). Moreover, 
computer-based systems have the unique capability of ensuring that guidance is followed 
by monitoring, evaluating, and providing feedback to individual authors at the point in the 
development cycle that is most useful to them. In addition, the utilization of Computer- 
based aids and other on-line author training materials would allow authors to integrate 
additional training into their schedules more easily. 

As stated earlier, the assessment of tri-service availabiity and utilization of author 
aids showed mutual interests and coordination. However, this coordination has been 
limited to an exchange of information rather than specific planning toward product 
transferability. Although some promising R&D efforts are being accomplished by each of 
the services, the authoring aids being developed are (1) service -specific, (2) still in the 
development stages, and (3) not ready for modifications that would meet Navy application 
needs. Although the Army is probably closest to having a usable author aid for ISD 
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program development, it is not clear at this point whether these aids will provide 
sufficient guidance to inexperienced authors. 

RECOMMENDATIONS 

1. To take advantage of those training design strategies that are known to be cost 
effective, those authoring aids already available, such as IQI and the Navy's Author and 
Training Course, should be modified if necessary and placed on-line as computer-based 
aids. 

2. NAVPERSRANDCEN should develop an author-aiding system that utilizes exist- 
ing aids and incorporates new ones as needed. The system should be transportable so that 
it can be widely used by groups developing Navy instruction in various locations. It should 
be available for tri-service utilization. 

3. IPDC staff members should be strongly encouraged to take advantage of 
available instructional technology courses and, whenever possible, time should be set aside 
for this training. 

'f. Coordination should be maintained with the appropriate tri-service agencies to 
facilitate interservice exchange, as well as eventual application of new authoring aids. 
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